DOC: Add AGENTS.md with basic type and docstring guidelines #62541
Conversation
|
Thanks for the PR. I don't have any experience with this so am open to experiences from other projects. The github link provides an example of an instructions file that is much more compact than what is provided here: Is there an advantage to this text copied from the guide versus something that would be more bulleted like the example? |
Thanks Will. My goal with the copy-paste was to avoid any controversy over wording. I'll ask copilot to re-write it in a way that is better suited for a copilot-instructions.md and update the PR with what it gives me ;) Separately, on Slack someone made me aware of AGENTS.md which is now natively supported by Github Copilot (among many other agents) so I will update this to be an agents.md instead. |
jzwick
changed the title
There was a problem hiding this comment.
I am neutral to these changes. But I think this file needs to be improved. I think the one from openai/agents.md is a great example.
| - doc/source/development/contributing_docstring.rst | ||
| - doc/source/development/contributing_documentation.rst | ||
| - doc/source/development/contributing.rst | ||
|
|
There was a problem hiding this comment.
| + doc/source/development/contributing_environment.rst | |
It needs instructions on how to build pandas from source. I am not sure if the model will be reliable if it reads all these files. I think it's best to create sections focusing on some aspects and link to the .rst files for additional information. The agents.md highlights:
- Project overview
- Build and test commands
- Code style guidelines
- Testing instructions
- Security considerations
There was a problem hiding this comment.
Thanks for the input. The intent of this PR is to "get the ball rolling" with a simple AGENTS.md, and to continue to expand upon it with subsequent PRs. For the first PR we wanted to stay focused on some simple guidelines around style (types and docstrings), and pointing to the existing contributing_*.rst
I agree that sections with more details for Build, Testing, and Security would be very valuable - we just prefer to have those contributed as separate PRs for more engaged conversation around each of those specifically.
There was a problem hiding this comment.
WRT whether to have the .rst linked at the top, or individually within each of the sections below, I'm open to what others have experience with. In one place seemed more maintainable (e.g. easier to see if one is missing, or renamed, where to update the reference).
|
Is the idea to encourage AI PRs? That’s the opposite of what we want. |
No, we are not trying to encourage AI PRs. IMO the motivation here is acknowledging the reality that many people are already developing with the help of AI agents, so we are trying to improve both the (human) developer's experience and the code quality that ends up in PRs. For example, if we have a good AGENTS.md with robust instructions around the Style Guidelines, then for folks who do leverage copilot, cursor, etc. hopefully that reduces the corrections made at PR review. |
There was a problem hiding this comment.
I still have some healthy skepticism of the effectiveness of this file. It would be nice to see the impact of this file when AI is used to help/solve a good first issue.
As an experiment, could you open an experimental PR that uses AI to address a good first issue while this file is present (and sharing all the transcripts from the AI, especially what rules it is following)? It would also be good to see another PR with a similar premise but other instructions exist e.g. .cursor/rules that may differ or conflict with this AGENTS.md
|
I think it will be difficult or impossible to measure the effectiveness of an AI solution, and that will lead to a level of subjectiveness with this in the future, but I'm also hesitant for either of those problems to be a blocker. AI tools are popular companions and learning tools, particularly for new contributors, so I am +1 to anything that promotes contribution |
I'm just interested if the AI solution even just uses this "Create a plan on how to solve $PANDAS_ISSUE and describe the conventions or rules to follow for this repository when implementing that plan." Gives any indication if it picked anything up from |
|
@mroeschke I think that's a fair question. I will do a bit of "experimenting" and report back with my findings. |
|
Thanks for being open to experimenting. For context, I am also not opposed to contributors using AI, but I (and other maintainers of larger open source projects I've spoken to) have noticed an increase of poor quality contributions in the past year which have the signs of AI use, which makes AI feels like a net negative experience from the maintainers side. So I'm hoping these "agent guidelines" can help AI follow conventions that a human would when contributing to pandas. |
|
There was a flood of AI slop PRs/issues a couple months ago but it seems to have calmed down. Any guesses as to why? Could it really be as simple as "because we asked people to stop"? |
|
@mroeschke @WillAyd here is a link to the results of my experimentation https://github.com/jzwick/pandas/blob/agentsmdexperiment/agentsmd_experiment.md TLDR; instruction files do have an impact, but (in my experience) only when the user prompts the agent to look at it. It only needs to be prompted once in the session, not with each instruction, even if a checkpoint is reverted. It didn't follow the instructions about loading other files or asking the Interested to see what you think. |
5 participants
doc/source/whatsnew/vX.X.X.rstfile if fixing a bug or adding a new feature.This PR adds as
AGENTS.mdfile, which is like "a README for agents: a dedicated, predictable place to provide the context and instructions to help AI coding agents work on your project." (more on AGENTS.md here)This PR started as a
copilot-instructions.md(see more on copilot-instrucitons.md files here) but the more genericAGENTS.mdwas suggested, which is supported not only by Copilot but also many other AI agents.I have initially populated the file with some simple guidance on Decision Heuristics, Type Hints and Docstrings, based on the existing guidance in the "Contributing" section of the project documentation. The expectation is that the content in this file would grow and become more comprehensive over subsequent PRs.